[SOUND] Hello everyone.
Welcome back to the Heterogeneous Parallel Programming class.
This is lecture 1.5. Introduction to CUDA
in we are in the memory allocation and data movement API functions part.
The objective of this lecture is to help you to learn the basic application
programming interface functions or API functions in CUDA host code.
API functions is a standard way
for industry to extend standard programming languages.
To support certain specialized functionalities.
In this case, the CUDA designers and NVIDIA
provided these API functions to help C programmers to
use throughput oriented devices such GPUs in a Heterogeneous
computing system The two types of API functions that we
will be looking at today are the device memory
allocation function and also the host device data transfer functions.
This slide shows the reviews that vector addition example, and we used this
example to illustrate data parallelism and today we're going to
use this example to show how you can easily a
standard sequential C program to do vector addition into a a
heterogeneous parallel piece of code to do the same functionality.
And just a reminder that each CUDA thread will
be doing, will be adding an element of A
and element of B and assigned to element of C.
So here we will look at a traditional C code for doing vector addition.
So, here we show that in the main function we in C we will be doing memory allocation
and then we'll do some IO read to reading A and B and then we will need to
determine that elements in A and B so [INAUDIBLE] N
So at some point, we would want to do vector addition.
So we call vecAdd function with four parameters.
We give the pointers to, to A pointer, pointers to A, B, C and then
the number of elements in these vectors. So, up here, we show a
simple sequential function for doing vector addition in C.
It matches the parameters that our main function is going to pass to it.
So it's going to say that, oh I
have three pointers are pointing to floating point arrays.
So it will be A, B and C and then it
expects a integer value that gives the number of elements in
A, B and C. Then there is going to be a loop, for loop
with a loop variable i and this for loop was sequentially goes through all the
elements of A, B add them up and assign to the corresponding C element.
So this is a sequential code, because the root will sequentially
visit all the A B elements and generate the corresponding C element.
Now we're going to show how we can systematically
convert this piece of code into a Parallel CUDA code.
The
this slide shows the outline of how we can change that vector add function
to be, to use the throughput-oriented GPU devices.
So instead of performing the, the actual complication, this function is actually
going to call a kernel function that will be executed on the device and in,
before calling that function the this function also needs to do some outsourcing
activity. It needs to copy data from the host memory
into the device memory, so that the device is ready to, to process the data.
And eventually after the device completes it's computation and
needs to copy the data's C vector back into the
host memory.
So when we look at the, the function there's also a header file
that you will need to include in order for that for this function to work properly.
This is the include file of cuda.h. So this is a line that you need to add to
your CUDA files in order for for the, the functions to be the host code to be
able to, to get access to all the API functions properly.
So, here we show three main parts in the host code.
The first part is to allocate device memory for A, B
and C and copy A and B to the device memory.
So this is illustrated in the top picture here where we have part
one that is copying data from the host memory to a device memory
after we allocated them in the device memory.
The second part is for the host code to launch the
kernel function, and this will be the topic for the next lecture.
And after the, the the kernel completes its execution, the host code
part three will copy C the output the result of the computation.
Back from the
device memory to the host memory.
And this is illustrated as part C in the picture.
In order to really understand what's going on with these API functions,
you need to have a good, a conceptual understanding of the CUDA memories.
And this picture is actually a simplified picture and we do
have all the CUDA memory, device memory types in this picture.
However, we're showing two important parts that you
will need to understand in order to understand
API functions and, and immediately in the next
lecture you also need to use, need to understand.
The registers in order to understand a simple piece of kernel code.
So the the simple way of looking at this
is that each device will have many, many threads.
So remember, these threads are actually virtualized Von Neumann processors.
So you can think about these threads as processors.
And each processor will have a set of registers.
And so these registers hold variables that are private to the thread.
And then all the threads will have access to a shared global memory.
So the, this is going to be also important.
In the kernel,
we're going to see that some of the accesses are
going to be to the shared global memory variables.
For the purpose of this lecture the more important part is
that the host code can actually allocate memory in the global memory.
And also request data copy from the global mem from the host memory to the global
memory and and vice versa that is from global memory back to host memory.
We will cover more memory types later in the, in
the subsequent lectures when we talk about localities and so on.
The first type of API functions that we're going to focus on is that CUDA
Device Memory Management API functions. Mainly this cudaMalloc and cudaFree.
cudaMalloc function allocates objects in a device global memory, right here.
And it would it will take two parameters, one is the address of a pointer to
the allocated object, and the other one is
the size of allocated object in terms of bytes.
For C programmers this is the pretty much
the same as the Malloc function because the
malloc function in C also requires the size
to be the allocated object in terms of bytes.
The difference that you will notice as a C programmer is that C malloc
actually returns a pointer value to the allocated object.
But here we actually passed the address of a pointer to the allocated pointer and
then the func, the allocation function is
going to extentiate the value into the pointer.
So that this is really a call by reference activity.
The reason why it's different is because
all the CUDA API functions return error code, which
we'll, we'll see in the next in a few slides.
So, because the return value is always the
error code then the only way that that cudaMalloc
function can systematically return a pointer to the allocated
object is by doing this call by reference convention.
The second function is cudaFree and cudaFree essentially frees the object
from the global memory so that the memory space can be, can be recycled.
And, it only takes one parameter which is a pointer to the free object.
Be very careful here the parameter in cudaFree is
the pointer to the free object whereas the parameter,
the first parameter to the cudaMalloc function is
the address of the pointer to the allocated object.
The second category of API functions that we'll be
using today is the host device data transfer API functions.
And this is mainly the cudaMemcpy or a function memcpy.
CUDA mem copy function, this is also fashioned after the C
mem copy function. It performs memory data transfer and
it requires four parameters, the first parameter is pointer to the destination.
And the second one is pointed to the source, the third one is number of
files to be copied, and the fourth one is the type or direction of the transfer.
And we typically will use predefined constants in CUDA to indicate this type,
and as we will see in the next slide. So the, the transfer
to the device by this function is asynchronous, meaning
that we can request one copy by calling cudaMemcpy.
But cudaMemcpy will return to the, will return right away, even before
the copy is complete, so that we can immediately request another memcpy.
And this is actually really very important when
we begin to utilize task level parallelism, and
so we're going to come back to this point later in the course.
So now we're, that we, we have
introduced the cudaMalloc, cudaFree, and cudaMemcpy were
ready to convert our vector addition example
code, host code, into the real host code.
So this function now is no longer just the outline.
We actually have all the statements that
implement that those parts, several part two.
So in part one now we have a declaration of DA,
DB and DC and these are pointers that
through the object allocated object in the device memory.
And so the first cudaMalloc function Is
going to allocate the device memory for vector A.
And as you can see, the size calculation is n times the size of float.
So each float in CUDA is four bytes, so
therefore, n is the number of elements in the vector.
So this gives us the size in terms of bytes, now we have cudaMemcpy and the
destination is the device memory, that's why it
is D_A, the sources in the host memory
that's why it's H_A.
The size gives the number of bytes and then
there is the pre-defined constant cudaMemCpy host to device.
And this constant is actually defined in that cuda.h
file that you included in the in the source file.
And we have once we allocated that memory, we can go and do the KUDA main copy,
we can do the same thing for Vector B. So we
allocate B and we do the copy from host B of B from host memory to device memory.
We allocate memory for C. We know, we don't need to copy from
host to C because C is the result of the computation, so the kernel
is going to generate the, all the values of C.
Part two remains remains a comment here, because we're going
to come back in the next lecture to complete this
part and then part three is to copy the data
the result from the device memory back into the host memory.
And we are going to use a constant CUDA mem copy device
to host, to indicate the direction of the copy, and after we're done
we can just go ahead and free A, B, and C from the device.
So this piece of code is, gives you the all the
code that needs to, that you need to have to allocate
memory, to copy data, in preparation for the kernel execution and
then copy the result data back and free up the memory.
In general, when we actually try to, to get performance out
of this kind of code, we cannot afford to copy
data back and forth before and after each kernel invocation.
So for real applications, we tend to have [UNKNOWN] data that just
reside in the device memory, and then we just keep launching Kernels to
perform computation on the device memory.
So but because this is the beginning of our course where it just showing a
very simple example and show you all the pieces of the, that can
be involved, so that you know exactly how to copy, allocate memory, you know
how to copy data from host to device, you know how to copy results
from device to host.
But in a real application, some of these steps may not be necessary, because the
data may just have already been residing in the memory, deep device memory.
Or in some cases, the result can stay on the device memory for future use.
They don't have to be copied back.
In practice what we have shown so far is that we just
go ahead and call it cudaMalloc function and that that's the job.
However, in practice I would like to
encourage you to always check for error conditions.
So here is what you really should have
done when you do, do call a cudaMalloc function.
You should declare a variable
of the type cudaError_t. And this is a predetermine,
defined type in the cuda API. And this is also from that cuda.h file.
And then you can declare a variable in this case we call that variable err.
So that when we call the cudaMalloc, we will
take the return value, assign that to err variable.
And this error
code is going to be checked.
You can check the error code and see if it is cudaSuccess.
Whenever the error code is cudaSuccess, that means
that the function has completed what you asked for.
In this case, cudaMalloc has successfully allocated the size of memory
and assigned that pointer to the object to the d_A pointer.
So that was a success.
However, if the error is not a success, then
you need to actually figure out what went wrong.
In most cases, the reason why there is an error condition in cudaMalloc is because
there's not enough device memory to satisfy this allocation request.
So a good way to bring out the error message is by calling
the cudaGetErrorString API function. This is a function that this also
provided as part of the CUDA API and this will convert error code into a stream.
That's human readable.
And then there just like the standard C functions, you
can use underscore, underscore, file,
underscore, underscore, and underscore, underscore,
line, underscore, underscore, to print out the position where the error happens.
And this is going to be the position with of your error checking right here.
So this is going to give you the line position
of this if, of this printf statement right here
and then you know, you can exit, then you
can exit a function with the exit failure code.
So that instead
of executing this code with the error already happened you can
actually ex, ex, exit the function and debug your function
and see why there's not a sufficient memory to satisfy this cudaMalloc request.
in, in the future slides, I'm still going to be showing cudaMalloc, cudaMemCpy
clauses all with this error code checking because that keeps the slide simple.
But when you do the lab assignments, I really would like
you to, encourage you to use this kind of error code sequence.
Even though it makes your code a lot bigger, in the long run,
it will save you a lot of time and a lot of a
lot of stress, because it will help you to catch any of these
errors so that you, it will be a lot easier for you to debug your program.
So, now we have completed a very quick introduction to the CUDA APIs that help
you to do memory allocation and data transfer, and we also showed a
error reporting String API function, so if you are
interested in understanding more about the CUDA
API functions please read chapter three of the textbook, thank you.